Plenoptic camera

ABSTRACT

One embodiment of the present invention provides a plenoptic camera which captures information about the direction distribution of light rays entering the camera. Like a conventional camera, this plenoptic camera includes a main lens which receives light from objects in an object field and directs the received light onto an image plane of the camera. It also includes a photodetector array located at the image plane of the camera, which captures the received light to produce an image. However, unlike a conventional camera, the plenoptic camera additionally includes an array of optical elements located between the object field and the main lens. Each optical element in this array receives light from the object field from a different angle than the other optical elements in the array, and consequently directs a different view of the object field into the main lens. In this way, the photodetector array receives a different view of the object field from each optical element in the array.

This application is a continuation of U.S. patent application Ser. No.11/398,403, filed Apr. 4, 2006.

COLOR DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

BACKGROUND

1. Field of the Invention

The present invention relates to cameras. More specifically, the presentinvention relates the design of a “plenoptic” camera, which capturesinformation about the direction distribution of light rays entering thecamera.

2. Related Art

Conventional cameras fail to capture a large amount of opticalinformation. In particular, a conventional camera does not captureinformation about the location of the aperture of the light raysentering the camera. During operation, a conventional digital cameracaptures a two-dimensional (2D) image representing a total amount oflight which strikes each point on a photosensor within the camera.However, this 2D image contains no information about the directionaldistribution of the light that strikes the photosensor. This directionalinformation at the pixels corresponds to locational information at theaperture.

In contrast, a “plenoptic” camera samples the four-dimensional (4D)optical phase space or light field and in doing so captures informationabout the directional distribution of the light rays. For example, see[Adelson92] Adelson, T., and Wang, J. Y. A. 1992, “Single lens stereowith a plenoptic camera,” IEEE Transactions on Pattern Analysis andMachine Intelligence 14, 2, February 1992, pp. 99-106. Also see [Ng05]Ng, R., Levoy, M., Bredif, M., Duval, G., Horowitz, M. and Hanrahan, P.,“Light Field Photography with a Hand-Held Plenoptic Camera,” StanfordUniversity Computer Science Tech Report CSTR 2005-02, April 2005. Thesepapers describe plenoptic/light-field camera designs based onmodifications to a conventional digital camera.

Referring to FIG. 1A, the system described in [Ng05] uses a microlensarray 106 comprised of about 100,000 lenslets which is placed a smalldistance (0.5 mm) from a CCD array 108. Each lenslet splits a beamcoming to it from the main lens 104 into (100) rays coming fromdifferent “pinhole” locations on the aperture of the main lens 108. Eachof these rays is recorded as a pixel, and the pixels under each lensletcollectively form a 100-pixel image. If we call this 100-pixel image a“macropixel,” then the plenoptic photograph captured by this camera willcontain approximately 100,000 macropixels. By appropriately selecting apixel from each macropixel, we can create conventional pictures takenwith a virtual pinhole camera. Moreover, by mixing such imagesappropriately, we can refocus images originally taken out-of-focus,reduce noise, or achieve other “light-field” effects, as described inthe papers above.

In the prototype described in [Ng05], a 16-megapixel sensor is used withan approximately 100,000 lenslet array to create a final output ofapproximately 300×300 macropixels, with one macropixel per lenslet. Themacropixel created by each lenslet comprises approximately 150 pixels.However, only about 100 of these pixels are useful because of poorquality of edge pixels caused by a problem which is referred to as“vignetting.” These 100 pixels which comprise each macropixel make thecaptured data equivalent to 100 conventional images, one for each choiceof the pixel inside a macropixel. The size of each picture produced byprocessing data from this camera is equal to the number of lenslets, andis hence 300×300.

Unfortunately, an image with only 300×300 pixels has insufficientresolution for most practical uses. The number of pixels can beincreased by increasing the number of lenslets and making them smaller.Unfortunately, the prior art cannot use the border pixels of each image.Note that a band of about 2 to 4 pixels along the border of themacropixel is lost depending upon whether the system is working with aGrayscale pattern or a Bayer pattern. When the image is small, these fewborder pixels comprise a large percentage of the image. For example, ina 10×10 color image, 4 pixels on each edge may be lost leaving only2×2=4 central pixels. In this case, 96% of the information lost! Becauseof this problem, the system described in [Ng05] cannot reduce the sizeof each microlens and the image under it. Consequently, the number ofmicrolenses, and hence the resolution of the image, is limited.(Currently, in a system that uses a 16-megapixel sensor, the number ofmicrolenses is limited to less than 100,000.)

Hence, what is needed is a method and an apparatus for increasing theresolution of a plenoptic camera without the above-described problems.

SUMMARY

One embodiment of the present invention provides a plenoptic camerawhich captures information about the direction distribution of lightrays entering the camera. Like a conventional camera, this plenopticcamera includes a main lens which receives light from objects in anobject field and directs the received light onto an image plane of thecamera. It also includes a photodetector array located at the imageplane of the camera, which captures the received light to produce animage. However, unlike a conventional camera, the plenoptic cameraadditionally includes an array of optical elements located between theobject field and the main lens. Each optical element in this arrayreceives light from the object field from a different angle than theother optical elements in the array, and consequently directs adifferent view of the object field into the main lens. In this way, thephotodetector array receives a different view of the object field fromeach optical element in the array.

In a variation on this embodiment, a given optical element in the arrayof optical elements includes: a lens; a prism; or a lens and a prism.

In a further variation, the lens is a negative lens with a negativefocal length.

In a further variation, the lens is an achromatic lens, and the prism isan achromatic prism.

In a variation on this embodiment, the photodetector array is aCharge-Coupled Device (CCD) array.

One embodiment or the present invention additionally includes aprocessing mechanism configured to process the different views of theobject field received by the photodetector array to produce a finalimage.

In a further variation, while producing the final image, the processingmechanism is configured to use the different views of the object fieldto adjust one or more of the following: a plane-of-focus for the finalimage; a viewing angle for the final image; or a depth-of-field for thefinal image.

In a further variation, while processing the different views of theobject field, the processing mechanism is configured to performview-morphing or interpolation operations between the different views toproduce additional views of the object field which appear to be gatheredfrom locations between the locations of the optical elements in thearray.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1A illustrates a prior art plenoptic camera.

FIG. 1B illustrates a plenoptic camera in accordance with an embodimentof the present invention.

FIG. 2 illustrates a layout for the additional lenses and prisms in aplenoptic camera in accordance with an embodiment of the presentinvention.

FIG. 3 illustrates an array of lenses and prisms in accordance with anembodiment of the present invention.

FIG. 4 illustrates an array of lenses and an array of prisms inaccordance with an embodiment of the present invention.

FIG. 5 illustrates an exemplary scene in accordance with an embodimentof the present invention.

FIG. 6 presents images of the exemplary scene taken through an array oflenses in accordance with an embodiment of the present invention.

FIG. 7 presents images of the exemplary scene taken through an array ofprisms in accordance with an embodiment of the present invention.

FIG. 8 presents images of the exemplary scene taken through both thearray of lenses and the array of prisms in accordance with an embodimentof the present invention.

FIG. 9A illustrates an image of the exemplary scene which is generatedso that both the foreground and background are in-focus in accordancewith an embodiment of the present invention.

FIG. 9B illustrates an image of the exemplary scene which is generatedwith the foreground in-focus in accordance with an embodiment of thepresent invention.

FIG. 9C illustrates an image of the exemplary scene which is generatedwith the background in-focus in accordance with an embodiment of thepresent invention.

FIG. 10 presents a flow chart illustrating how light is directed withina plenoptic camera in accordance with an embodiment of the presentinvention.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled inthe art to make and use the invention, and is provided in the context ofa particular application and its requirements. Various modifications tothe disclosed embodiments will be readily apparent to those skilled inthe art, and the general principles defined herein may be applied toother embodiments and applications without departing from the spirit andscope of the present invention. Thus, the present invention is notlimited to the embodiments shown, but is to be accorded the widest scopeconsistent with the claims.

The data structures and code described in this detailed description aretypically stored on a computer-readable storage medium, which may be anydevice or medium that can store code and/or data for use by a computersystem. This includes, but is not limited to, magnetic and opticalstorage devices such as disk drives, magnetic tape, CDs (compact discs),DVDs (digital versatile discs or digital video discs), or any devicecapable of storing data usable by a computer system.

Overview

In the system described in [Ng05], the size of each image for eachmacropixel is limited by the design of the camera. The present inventionprovides a new, easier-to-construct camera, which gives the designermore flexibility in the trade off between the number of views and thesize of each view image. In particular, one embodiment of the presentinvention can capture a small number (10) of high-resolution images,whereas the system described in [Ng05] captures a large number (100) oflow-resolution images.

Hence, one embodiment of the present invention provides a design inwhich a small number of conventional lenses are placed in front of themain lens of the camera, instead of placing a much larger number ofmicrolenses behind the main lens as is disclosed in [Ng05], therebymaking plenoptic cameras much easier to build.

Note that the present invention makes it possible to reduce the numberof effective images captured (from 100 in the prior art, down to 20, oreven to 10) without loss of quality at the edge pixels. This is a majorproblem for the system disclosed in [Ng05], which limits imageresolution to 300×300 macropixels. Using the same type of opticalsensor, the present invention can achieve a significantly higherresolution for the final image (for example, five times the resolution).In other words, the present invention provides the flexibility to tradedetail in the 3D information for detail in the 2D images. For simplescenes, composed of a few surfaces in 3D, a few images are sufficient tocapture the complete 3D detail. (For example, the human visual systemuses only two images.) This speaks in favor of reducing the number ofimages, because 100 images is probably excessive, and 10 or 20 images isprobably more appropriate. Embodiments of the present invention makesthis possible, while it was not possible in the prior art.

Instead of locating the array of lenses behind the main lens, as in theprior art system illustrated in FIG. 1A, the present invention placesthe array of lenses 114 in front of the main lens 116 as is illustratedin FIG. 1B. More specifically, one embodiment of the present inventionachieves higher-resolution results by placing 19 lenses/prisms in frontof the main lens instead of placing 90,000 lenses behind the main lens.

In the embodiment of the present invention illustrated in FIG. 1B, anarray of (about 10 to 100) lenses 114 and prisms 112 is placed 200 to500 mm in front of the main lens 116 of a conventional camera. Note thatthe ratio of (width of array of lenses)/(distance to main lens) isideally equal to the f-number of main lens 116.

Each lens is coupled with a corresponding achromatic prism, wherein theprism has a different angle for different lenses, depending on locationof the lens. In particular, each prism is chosen to produce an angulardeviation which is equal to the angle at which the main camera lens seesthat prism. In this way, all prisms create images of the same objectfrom the scene. Note that it is not essential to achieve precision withthese prism angles and the arrangement of the prisms because smallerrors do not influence the final image quality. However, big errorsshould be avoided to ensure that pixels are not wasted due to randomshifts of images which create gaps and/or overlaps.

In one embodiment of the present invention, all of the lenses have thesame negative focal length, for example −100 mm. This focal lengthcontrols the field of view. Note that it is important for all lenses tohave the same focal length if we want good focusing.

The main lens 116 of the camera is focused on an array of virtual imageswhich appear in front of the negative lenses. Note that each lens/prismin the array receives light from image field 110 from a different anglethan the other lenses/prisms in the array, and consequently directs adifferent view of the image field 110 into the main lens 116 of thecamera. In this way, CCD array 118 captures an array of pictures inwhich each picture provides a different view of the object field from adifferent lens/prism in the array.

The array of pictures captured by CCD array 118 is processed byprocessing device 120 to produce a final image 122. (Note thatprocessing device 120 can be integrated into the camera or can belocated outside of the camera.) By mixing these images appropriately,processing device 120 can achieve various “light-field” effects, such asrefocusing an image, reducing noise, adjusting the viewing angle, andadjusting the depth-of-field for the final image. (For a description ofthe details of some of these light-field effects, please see [Adelson92]and [Ng05], also see U.S. Pat. No. 5,076,687, entitled “Optical RangingApparatus,” by inventor Edward H. Adelson.)

In one embodiment of the present invention, processing device 120 isadditionally configured to perform view-morphing or interpolationoperations between the different views to produce additional views ofthe object field which appear to be gathered from locations between thelocations of the lenses/prisms in the array. In this way, the presentinvention can produce a large number of images (100) using a smallernumber of lenses (20). (This type of view-morphing operation isdescribed in U.S. Pat. No. 6,351,269, entitled “Multiple ImageMorphing,” by inventor Todor Georgiev.)

Note that generating these additional views greatly opens up the designspace because the resulting system generates a large number of“high-resolution” images. This is an improvement over the systemdescribed in [Ng05], which captures a large number of “low-resolution”images.

Exemplary Embodiment

In an exemplary embodiment of the present invention, the array of lenses114 contains 19 lenses (with f=−100 mm), and 18 prisms which arearranged in a hexagonal pattern as is illustrated in FIGS. 2 and 3. Notethat the central lens has no prism because it is located on the mainaxis of the camera. The main camera lens has f-number f/2, whichcorresponds to 14 degrees. To accommodate this f-number, the prisms arechosen with deviation angles of 4 degrees, 7 degrees and 8 degrees as isillustrated in FIG. 2. In the exemplary embodiment, the lenses havediameter 25 mm and the total width of the array of lenses 114 is 125 mm.Moreover, the array is positioned at distance of 250 mm from main lens116. (Note that this distance can be adjusted.) With a 16-megapixel CCDarray, this embodiment is able to capture final images of about 600×600pixels, which is 4 times better than the camera described in [Ng05] forthe same camera resolution.

Note that a tube, which looks like a telephoto lens can extend from themain lens to the array of lenses to prevent light from entering thesystem sideways and forming reflection spots on the prisms and lenses.

Lenses and Prisms

One embodiment of the present invention can operate using prisms onlyand using lenses only, but it is preferable to use both lenses andprisms. To illustrate this, a number of pictures have been taken throughthe array of 7 negative lenses and the corresponding array of 6 prismsillustrated in FIG. 4. These lenses and prisms are used to captureimages of an exemplary scene which appears in FIG. 5.

FIG. 6 illustrates images of the exemplary scene which are taken throughthe array of lenses only. Note that these images are shifted withrespect to each other and do not capture identical areas of the scene,although there is a small area of the scene near the cap of the tubewhich appears in all of the images.

FIG. 7 presents images of the exemplary scene which are taken throughthe array of prisms only. Note that these prisms shift the images so thesame part of the scene is captured in each image. However, the resultingfield of view is quite narrow.

Finally, FIG. 8 presents images of the exemplary scene which are takenthrough both the array of lenses and the array of prisms. Note that theprisms shift the images so that all the images are centered and thelenses expand the field of view. Also note that each two images form astereo pair.

By using a negative lens instead of a positive lens, the plane where theimage is formed is further away from the camera. This makes theresulting system more compact, because it allows the array of lenses tobe closer to the main lens.

Generating a Resulting Image

As mentioned above, the present invention can achieve various“light-field” effects, such as refocusing, reducing noise, adjusting theviewing angle, and adjusting the depth-of-field for the image. Forexample, FIGS. 9A-9C illustrates how one embodiment of the system canvirtually focus on different image planes after a picture has been takenin accordance with an embodiment of the present invention. In FIG. 9A,the depth-of-field of the image is large, so both the bottle in theforeground and the tube in background are in-focus. In FIG. 9B, thedepth-of-field is reduced and the focal plane of the image is set to benearer to the camera, so the bottle in the foreground is in-focus, whilethe tube in the background in out-of-focus. In FIG. 9C, the focal planeof the image is set to be farther from the camera, so the tube in thebackground is in-focus, while the tube in the foreground isout-of-focus.

Light Flow

FIG. 10 presents a flow chart illustrating how light is directed withina plenoptic camera in accordance with an embodiment of the presentinvention. First, light is received from objects in an object field atan array of optical elements located between the object field and themain lens of the camera (step 1002). Each optical element in this arrayreceives light from the object field from a different angle, andconsequently directs a different view of the object field into the mainlens.

Next, light is received from the array of optical elements at the mainlens which directs the received light onto an image plane of the camera(step 1004).

Then, light is received from the main lens at a photodetector arraylocated at the image place of the camera (step 1006), wherein thephotodetector array receives a different view of the object field fromeach optical element in the array.

Finally, the different views of the object field which are received bythe photodetector array are processed to produce a final image (step1008).

The foregoing descriptions of embodiments of the present invention havebeen presented only for purposes of illustration and description. Theyare not intended to be exhaustive or to limit the present invention tothe forms disclosed. Accordingly, many modifications and variations willbe apparent to practitioners skilled in the art. Additionally, the abovedisclosure is not intended to limit the present invention. The scope ofthe present invention is defined by the appended claims.

1-20. (canceled)
 21. A camera, comprising: a main lens which receiveslight from objects in an object field and directs the received lightonto an image plane of the camera; a photodetector array located at theimage plane of the camera, which captures the received light to producean image; and a multidimensional array of optical elements locatedbetween the object field and the main lens, wherein each optical elementin the array receives light from the object field from a different anglethan the other optical elements in the array of optical elements andconsequently directs a view of the object field into the main lens,whereby the photodetector array receives a different view of the objectfield from each optical element in the multidimensional array of opticalelements; wherein each different view of the object field is received ata separate location on the photodetector array to produce amultidimensional array of different views of the object field at thephotodetector array.
 22. The camera of claim 21, wherein a given opticalelement in the array of optical elements includes: a lens; a prism; or alens and a prism.
 23. The camera of claim 21, wherein one or more of theoptical elements comprise negative lenses with negative focal length.24. The camera of claim 21, wherein a plurality of the optical elementscomprise: achromatic lenses of a same focal length; or an achromaticprism.
 25. The camera of claim 21, wherein the photodetector array is aCharge-Coupled Device (CCD) array.
 26. The camera of claim 21, furthercomprising a processing mechanism configured to process the differentviews of the object field received by the photodetector array to producea final image.
 27. The camera of claim 26, wherein while producing thefinal image, the processing mechanism is configured to use the differentviews of the object field to adjust one or more of the following: aplane-of-focus for the final image; a viewing angle for the final image;and a depth-of-field for the final image.
 28. The camera of claim 26,wherein while processing the different views of the object field, theprocessing mechanism is configured to perform view-morphing orinterpolation operations between the different views to produceadditional views of the object field which appear to be gathered fromlocations between the locations of the optical elements in the array ofoptical elements.
 29. A method for gathering light, comprising:receiving light from objects in an object field at a multidimensionalarray of optical elements located between the object field and a mainlens of a camera, wherein each optical element in the array of opticalelements receives light from the object field from a different anglethan the other optical elements in the array of optical elements andconsequently directs a different view of the object field into the mainlens; receiving light from the array of optical elements at the mainlens which directs the received light onto an image plane of the camera;and receiving light from the main lens at a photodetector array locatedat the image plane of the camera, wherein the photodetector arrayreceives a different view of the object field from each optical elementin the multidimensional array of optical elements, wherein eachdifferent view of the object field is received at a separate location onthe photodetector array to produce a multidimensional array of differentviews of the object field at the photodetector array.
 30. The method ofclaim 29, wherein a given optical element in the array of opticalelements includes: a lens; a prism; or a lens and a prism.
 31. Themethod of claim 29, wherein one or more of the optical elements comprisenegative lenses with negative focal length.
 32. The method of claim 29,wherein a plurality of the optical elements comprise: achromatic lensesof a same focal length; or an achromatic prism.
 33. The method of claim29, wherein the photodetector array is a Charge-Coupled Device (CCD)array.
 34. The method of claim 29, further comprising processing thedifferent views of the object field received by the photodetector arrayto produce a final image.
 35. The method of claim 34, wherein producingthe final image involves using the different views of the object fieldto adjust one or more of the following: a plane-of-focus for the finalimage; a viewing angle for the final image; and a depth-of-field for thefinal image.
 36. The method of claim 34, wherein processing thedifferent views of the object field involves performing view-morphing orinterpolation operations between the different views to produceadditional views of the object field which appear to be gathered fromlocations between the locations of the optical elements in the array ofoptical elements.
 37. An imaging system, comprising: a main lens whichreceives light from objects in an object field and directs the receivedlight onto an image plane; a photodetector array located at the imageplane, which captures the received light to produce an image; amultidimensional array of optical elements located between the objectfield and the main lens, wherein each optical element in the arrayreceives light from the object field from a different angle than theother optical elements in the multidimensional array of optical elementsand consequently directs a view of the object field into the main lens,whereby the photodetector array receives a different view of the objectfield from each optical element in the array of optical elements;wherein each different view of the object field is received at aseparate location on the photodetector array to produce amultidimensional array of different views of the object field at thephotodetector array; and a processing mechanism configured to processthe different views of the object field received by the photodetectorarray to produce a final image.
 38. The imaging system of claim 37,wherein a given optical element in the array of optical elementsincludes: a lens; a prism; or a lens and a prism.
 39. The imaging systemof claim 37, wherein while producing the final image, the processingmechanism is configured to use the different views of the object fieldto adjust one or more of the following: a plane-of-focus for the finalimage; a viewing angle for the final image; and a depth-of-field for thefinal image.
 40. The imaging system of claim 37, wherein whileprocessing the different views of the object field, the processingmechanism is configured to perform view-morphing or interpolationoperations between the different views to produce additional views ofthe object field which appear to be gathered from locations between thelocations of the optical elements in the array of optical elements.